{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2b6b7d0-dd86-4275-be30-c83df4b925ea",
   "metadata": {},
   "source": [
    "# Chat Completions for Local LLM\n",
    "\n",
    "There may be times when you want to use Guidance with a local LLM and a chat-completion API (as an alternative to using the Guidance DSL). For this, we provide an **experimental** API on the `Engine` class. This is used internally by the `Model` class as a uniform interface to LLMs loaded by both Tranasformers and LlamaCpp.\n",
    "\n",
    "We start by loading our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1dcd8597-9429-4cd3-8259-0c82139fb210",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4541f938e57240b39ecb1b2f5ea0e46e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "gpustat is not installed, run `pip install gpustat` to collect GPU stats.\n"
     ]
    }
   ],
   "source": [
    "import guidance\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "gguf = hf_hub_download(\n",
    "    repo_id=\"microsoft/Phi-3-mini-4k-instruct-gguf\",\n",
    "    filename=\"Phi-3-mini-4k-instruct-q4.gguf\",\n",
    ")\n",
    "\n",
    "# Define the model we will use\n",
    "# lm = guidance.models.LlamaCpp(gguf, n_gpu_layers=-1)\n",
    "lm = guidance.models.Transformers(\"microsoft/Phi-3-mini-4k-instruct\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9957c1-e055-4e98-874d-84c2499fdbc9",
   "metadata": {},
   "source": [
    "Now, set up a simple conversation as one would for a remote endpoint such as the OpenAI API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "eb0a9cbc-ae9d-4727-8c31-28ff7e7daee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\" : \"user\",\n",
    "        \"content\": \"Tell me about yourself\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cd6266a-3968-4008-9764-71c85af2065a",
   "metadata": {},
   "source": [
    "Define the Lark grammar which we wish to apply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "799135a0-5221-4182-91e0-98d335bb74c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "grammar = \"\"\"%llguidance {}\n",
    "\n",
    "start: \"My name is \" name \" and my motto is \" motto\n",
    "name[capture=\"name\", temperature=1.0]: NAME\n",
    "motto[capture=\"motto\", temperature=0.7]: MOTTO\n",
    "NAME: \"Phi-3, the Magnificent\"\n",
    "    | \"Phi-3, the Terrible\"\n",
    "    | \"Phi-3, the Great\"\n",
    "    | \"Phi-3, the Conqueror\"\n",
    "MOTTO: \"Look on my works ye mighty, and despair\"\n",
    "    | \"Apres moi, le deluge\"\n",
    "    | \"Alea iacta est\"\n",
    "    | \"Apres moi, le table\"\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2249736-7a9f-4de9-8a84-fac271e37e4b",
   "metadata": {},
   "source": [
    "Grab the `engine` object out of the Model. Note that this is only going to work for local models (basically, Models contain Interpreters, but only the local Interpreters then contain an engine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0c2ef1fe-eb9f-4412-9941-d74de7c76301",
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = lm._interpreter.engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99b9945d-6dcc-4f18-8440-4308cb3011cd",
   "metadata": {},
   "source": [
    "Call the experimental `chat_completion` method, supplying the conversation, grammar, and optionally some tools. These are all rendered with the model's chat template before inferencing. In response, we get the completion text and a dictionary of captures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ec01b076-42ef-46f1-aaf7-f009ae68c491",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\riedgar\\source\\repos\\guidance\\guidance\\models\\_transformers.py:522: UserWarning: Cache is too small. Resetting cache (no method implemented to resize cache for type <class 'transformers.cache_utils.DynamicCache'>).\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "completion, captures = engine.chat_completion(messages, grammar, tools=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f470b24-4791-4fa5-88be-7f86fad820b3",
   "metadata": {},
   "source": [
    "See the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9f0c9eb4-7aa6-4b4f-a2b4-c0edf7f53dfa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "completion='My name is Phi-3, the Magnificent and my motto is Apres moi, le deluge'\n"
     ]
    }
   ],
   "source": [
    "print(f\"{completion=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dd1a6478-51b0-449f-b263-e007ef4c82b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name : Phi-3, the Magnificent\n",
      "motto : Apres moi, le deluge\n"
     ]
    }
   ],
   "source": [
    "for k, v in captures.items():\n",
    "    print(f\"{k} : {v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f59f64-c4db-4242-8b5c-5c65bc750bba",
   "metadata": {},
   "source": [
    "There is also a streaming version of this API. Again this is an **experimental** API and is subject to change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d370da11-d82b-47aa-ba09-631ee327dfb6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
